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ABSTRACT 

This paper provides practical information to help 
experiential educators from varying backgrounds effectively use 
program evaluation. Program evaluation uses specific and systematic 
procedures that result in findings that are useful to decision makers 
for achieving program goals. The purpose of any program review 
process must be clear, including accountability, program improvement, 
information dissemination, and theory generation. This paper outlines 
a model of evaluation that is both simple to use and easily adapted 
to a variety of situations and purposes. This model includes: (l) 
establishing terms of reference; (2) gathering data; (3) developing 
an interim report; (4) developing an action plan; (5) presenting a 
final report; (6) implementing the action plan; and (7) following up. 
A list of data-gathering tools is included. Three factors affecting 
the outcome of any review are: the purpose of the program review, who 
conducts the program review, and who will use the results and when. 
Other suggestions for conducting program reviews include anticipating 
roadblocks to evaluation before the review process begins, 
structuring a time frame for the review process, and developing terms 
of reference for data gathering. An example of terms of reference 
designed for the review of a school physical education program is 
provided. (LP) 
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The primary purpose of evaluation is not to prove but to improve 

Stufflebeam & Guba 

The educational innovator usually has to design an evaluation 
witliout much professional advice, conduct it with limited 
resources, and defend it without the advantage of comparisons to 
conventional approaches that have been similarly evaluated. 

5. Hamilton 

Introduction 

Program evaluation is an oft-times delicate and intimidating subject. It is seen by many as something 
that someone else *does to you'. In this presentation we want to provide a brief background 
regarding the various purposes and methods that evaluation can serve but primarily we want to 
provide practical information and experience to help experiential educators from a wide spectrum 
of backgrounds use evaluation as a means to chart tlieir future. 

Wisely charting the future requires an assessment of past and present and an examination of 
assumptions about what is likely to occur in the future. A program evaluation can help address such 
questions and consequently inform planning so that it is based on what has been discovered, what 
has been confirmed and what is anticipated. The overall purpose of a well constructed and conducted 
evaluation is serWce; service to the organization, its stakeholders and its clients. Evaluation serves 
the needs of a program by providing information that is useful for making practical choices 
regarding quality and effectiveness. 

Our goal in this workshop is not to provide a crash course in research-oriented educational 
evaluation but to provide some tools, germinal experiences and additional resources that can help 
you as an experiential educator develop and conduct a program review that is thorough, realistic and 
useable. 
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What is Evaluation? 

Evaluation in its many forms has a long history. From time immemorial people have judged die 
quality and effectiveness of things and have assigned value accordingly. Methods of hunting, ways 
of treating illness and injury, training for sport, approaches to human interaction, art and wilderness 
leadership have all been hot topics of formal or informal debate and evaluation. Today we talk of 
evaluating outdoor leaders as a prerequisite to givmg them a stamp of approval such as certification. 
Such behavior is hardly new. Over 4,000 years ago the Chinese evaluated key public officials every 
three years to determine their fitness for office. As individuals involved in experiential education 
programs we are concerned about the effectiveness of what we do. So^'^^mes we need to justify 
the value of our efforts to outsiders. Evaluation is one way of establish; ^ die worth of something 
such as a novel educational approach. But what exactly is evaluation? 

Evaluation as we will be addressing it in this workshop is a systematic, 'formal' assessment of the 
quality and effectiveness of a particular program . However, a few additional words of clarification 
are in order. Because of an abundance of negative baggage associated with the term "evaluation" 
there has been a widespread shift to the use of other terms which have a similar meaning but a less 
threatening demeanor. Review and assessment are several of the evaluation synonyms that have 
gained popular usage. For many, the term evaluation conjures up the image of Dagwood Bumstead's 
boss Mr. Dithers, giving evaluative feedback at the top of his lungs. The purpose of such 
'evaluation' is to embarrass and humiliate. The idea, so the reasoning goes, is to motivate 
individuals or programs to do a better job by showing them how much room there is for 
improvement. Generally, however, this deficiency approach to evaluation is just plain demoralizing. 
Evaluation is not only concerned with pointing out what could be done better it also strives to 
highlight where a program's strengths lie. For our purposes we will use the term program review 
interchangeably with evaluation to refer to the assessment of a program using specific and systematic 
procedures which result in findings that are usefiil to decision makers for the purpose of helping 
them better shape and achieve their goals. 

Purposes of Evaluation 

Sergiovanni (1987) has proposed diree basic categories to explain die reasons why evaluations should 
be undertaken and what primary purposes they can serve. 

• Quality control - insuring program goals are being achieved in a manner consistent widi 
program values and that outcomes or inputs which are unintended but positive and important 
are recognized. 

• Professional development - helping individuals involved in planning and delivering the 
program to grow personally and professionally by continually expanding and enhancing their 
own knowledge, skills and attitudes. 

• Motivation of individuals involved in die program - building and nurturing motivation and 
commitment to die program and its goals. This includes die program's ability to take care 
of its employees. 

Sergiovannl's categories recognize die fact diat diere are a great many reasons to conduct a program 
review and that understanding the purpose of a review is the first step in determining its 
effectiveness. 
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Experiential educators conduct program reviews to answer different questions; field supervisors want 
to know what program elements are effective and which elements can be improved on; field staff 
want participant feedback on their effectiveness as teacher, facilitator and outdoor leader; the board 
of directors is concerned about the thoroughness and effectivenc^ss of safety procedures for both 
program facilities and field operations; the administrator is curious how their program measures up 
to other similar programs in tiie field. Like-wdse there are those who may want to discredit a 
program or conversely to promote it; both see a review as a means of justifying tiieir own particular 
action or position. 

Because of widely disparate intentions it is imperative for all stakeholders concerned to be sxire of 
the real purpose of any program review process. This requires personal and political acumen. Ideally 
the results of a review will inform and influence decision makers, but in reality if the findings are 
not in keeping with the opinions of those who requested or required the review the results may be 
convenientiy misplaced for a few millennia and the desired course of action taken irrespective of the 
review results. 

It is equally important to be sure of purpose when requesting or conducting a review. What do you 
want to accomplish or find out? How will the program review results help you in this regard? What 
form of finished report will be easiest to use? Who will see the finished review? What action will 
be taken and who will take responsibility for it? Poor definition of purpose is a recipe for 
misunderstanding, resentment, defensiveness and general disregard for the findings. Some of the 
purposes for a program review are categorized below. 

Accountability to: 

• program staff 

• participants/clients 

• funding sources (government, private agencies, taxpayers, sponsors) 

• governing bodies (government, schoolboards, professional groups) 

• board of directors 

• parent organizations 

Program Improvement Through: 

• revising program goals and objectives 

• increased recognition and understanding of successful strategies already in use 

• assessment of intended or unintended outcomes (performance) 

• more efficient and economical operation 

• establishing or improving community contacts 

• identification of staff training needs & desires 

• improving logistical procedures 

• improving equipment selection, use and maintenance 

• updating program content information or delivery methods 

Information Dissemination 

• networking with peers involved in the review process 

• highlighting the effectiveness of experiential education to an audience beyond ellow 
practitioners 

• lobbying for the inclusion of experiential education in more traditionally styled 
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Theory Generation 

• developing an understanding of the experiential education process instead of looking only 
at its outcomes 

Methods of Evaluation 

There are many reasons for conducting a program review and there are many ways of going about 
the review itself. Many of the formal and informal evaluation methods and data gathering tools used 
in experiential education have their roots in tiie field of educational evaluation. There are a great 
many comprehensive evaluation methods that can be used to conduct a review. Some are quite 
simple while others are unlikely to be used by any other than a trained educational evaluator and 
a specialist support team. There is a definite difference between statistics oriented, large scale 
evaluations focused on nation-wide educational programs atxl conducted by trained evaluation teams 
and the less formal evaluation done by a program administrator of their own program. The latter 
however is closer to reality for most experiential educators. The question becomes one of doing the 
best possible program review with the available resources. The hardest to come by resources are 
usually time, money and trained evaluators. For this reason we have chosen to focus on a model of 
evaluation that is both simple to use and easily adapted to a variety of situations and purposes (see 
the accompanying model The Program Review Process) and which incorporates data gathering tools 
from several of the more popular and widely used evaluation methods. It is a process we use 
personally and which we find practical with a wide spectrum of programs and people. 
Having stated our own preference and focus it is important to note that there are an amazing number 
of evaluation models out there and the curious amongst you are encouraged to consult the second 
edition of W. James Popham's witty and thorough text Educational Evaluation for more information. 
We have also included a References section (an expanded resource list will be presented at the 
workshop itself) that provider further materials for investigation. 

Data Gathering Tools 

Models are the large scale methods of program evaluation. The smaller scale methods are usually 
referred to as data gathering iools. They are simply a way of collecting information that can be 
analyzed. The review process may include many data gathering tools, tools which come in all shapes 
and sizes. Many of those listed below will probably be familiar to you in one form or another. 

• participant observations 

• review of documentation 

• performance tests 

• criterion referenced tests 

• questionnaires 

• interviews (recorded, videoed or written), individual (participants, instructor-teachers, 
administrators etc.) group 

• instructor/teacher self assessments 

• journal analysis 

• site assessment 

• at-task analysis 

• instructional strategy analysis 

• "in-flight" corrections 
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The Program Review Process 

simplied version 
as per Cooney & Hendricks 
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The Program Review Process 

expanded version 
as per Cooney & Henricks 
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Conductors and Consumers of Evaluation 



Much of tlie value of a program review has to do witli who conducts the review and who 
requests or requires it. 



Even when the purpose of a program review has been made explicit the possibilities for hidden 
agendas are rife. No reviewer is a perfectly unbiased observer but the quality of the reviewer is 
mirrored by the value of the evaluation results. Especially when reviews are conducted internally 
but their results circulated externally there is the possibility of reviewers not seeing or reporting 
those things that might reflect unfavorably on the program. 

Three factors will affect the outcome of any review: 

1. The purpose of the program review - if the stated purpose of a program review is 
ambiguous or a blatant misrepresentation of reality the results are likely to be worthless 
and unusable. Specific and manageable terms of reference generated collaboratively are 
the ideal. 

2. Who conducts the program review - it is crucial to identify reviewers who can 
accomplish the stated purpose in an efficient and effective manner. They may be drawn 
from within the program itself, from peers outside the program or from some agency 
such as a department of education or a professional group (Hamilton, 1980; Duckett, 
Strother & Gephart, 1982). 

3. Who will use the results and when - it is important that the reviewers understand not 
only the purpose of the review but also who it will be usexl by anu when. If the final 
report will be used by the board of directors to make decisions regarding capital 
expenditures then they must receive the final draft with enough time to review it prior to 
their annual meeting rather than receiving it three days after the meeting is over. 
Likewise if the final report is to be used by paperwork swamped administrators it must 
be formatted in a way that makes it quick and easy to read but which includes enough 
detail to be usefiil for decision making. 

Roadblocks in Evaluation 

There are many potential roadblocks to evaluation. Identifying the most imposing of these before 
the review begins will go a long way toward alleviating problems down the road. Try and 
address the major roadblocks when designing tlie review. Many times tiiis is a matter of 
inclusion; including those individuals or stakeholders directly affected by the review so as to give 
them ownership and input. There is much less resistance to and fear of a review which people 
have had a personal hand in developing. Here is a starter list of roadblocks previous groups and 



conductors 
program director 
director and staff team 
outside experts 
peers in the field 
professional evaluator 



staff 
administration 
board of directors 
funding sources 
government 



consumers 
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individuals have identified as problems for them. 

L Time - When can I do it? I'm too busy. 

2. Disruption of progiamming. 

3. Fear among staff about intent - hidden agendas. 

4. What is the pay-off? What is die cost? 

5. Who will benefit from the evaluation? 

6. Others - add your own. 

Timelines 

Timelines provide a structure to help insure that a program review doesn't turn into a ten year 
project. In addition to a specific time frame for completion make sure that flie individual or 
group responsible for a particular task is identified. Develop the time frame with careful 
reference to the programs operating time frame. For instance when looking at a mountain based 
adventure education program don't schedule field observations for the first week of courses when 
staff are still getting used to one another and trying to determine how they "fit" within the 
program. Once again check your plans with the relevant stakeholders (field staff, administrators, 
teachers etc.) to make sure you haven't overlooked anything major. Listed below is a sample 
time frame. 





Staff ratify terms of reference, evaluation process and timeline. 


April 15 




Review team established based on program input 


April 30 




Meeting with program members (staff, administration, board etc,) 


May 30 




Staff completes self-appraisal 


June 15 




On-site visits and data gathering July 1 


- August 1 




Review team meeting to develop draft report 


August 5 




Draft report presented to program 


August 15 




Review of draft report and development of action 
plan by program 


August 20 




Discussion of report, necessary changes made and action 
plan included to yield final report which is distributed 


Sept 15 


* 


This is a suggested timeline. Dates may change depending upon availability of 
teachers, instructors, reviewers etc. 
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Terms of Reference 

The terms of reference are data gathering guidelines which provide agreed upon parameters for 
the review process. They are also a statement of purpose in more specific terms. They specify 
what will be evaluated and by what standards. This can go a long way toward making the 
reviewer's job more manageable and as a result making the final report more relevant and 
useful. The terms of reference showh1>elow were designed for the review of a school physical 
education program which placed an emphasis on outdoor pursuits. 

A. Program 

Are the 'thrusts' of the physical education curriculum guides in place? e. g.; 

• are tiie various dimensions being offered (aquatics, dance, fitness, gymnastics, individual 
activities and outdoor pursuits)? What is offered? 

• is the movement approach being addressed at the elementary level? How is it implemented? 

• is the levels approach being used at the secondary level? How is it implemented? 

B. Planning 

• What statements are made regarding philosophy, goals, objectives, outcomes? 

• What instructional planning is in place? e.g. year, unit, daily documents, teaching resources. 

C Balance 

• What is the balance of time for activities offered in each dimension? 

D. Instructional Strategies 

• What strategies re being used? Are they effective? 

E. Student Evaluation 

• What are the criteria? Are they appropriate? How are they communicated? How are they 
assessed? 

F. Intratnurals 

• What activities, clubs, special days and self-directed activities are being offered? 

G. Professional Development 

• What plans exist? How are PD experiences coordinated between divisions? 

H. Work Environment 

• What are the arrangements for facilities, office space, §howers etc.? 
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Program Review Tips and Strategies 

1. Clearly state the purpose of the review. Set specific objectives. What is going to be reviewed 
and how? 

2. Keep the number of things being loolced at to a manageable level. 

3. Selection and training of evaluatinn team members is important. Training should focus on the 
skills of planning, c^immunication, observation, analysis, problem-solving and conflict 
resolution in addition knowledge about and experience with the content area, 

4. Feedback of a gene' ^lature is useless, e.g. ''lacks organization\ Be specific enough to meet 
the client*s needs gi^mg observed examples to illustrate your points, 

5. Insure that the final report is presented in a format and mianner tliat is useful to the client, 

6. Timing is important. This is tnie for planning observations, presenting feedback and scheduling 
release of the reports to cile only a few examples, 

7. Involve staff in the process of selecting the terms of reference aixl give them a say in the steps 
of the review, 

8. Evaluators should strive to make the review process as collaborative between evaluators and 
program members as possible. 

Conclusion 

The Program Review Process is a valuable and realistic means by which programs can assess 
specific aspects of their operation and thereby help chart tlieir course into the future. There are many 
puiposes for doing a program review and many individuals or combinations of individuals wh nay 
serve as conductors of the review process. The conductors may come from within the organi/^ation 
itself, from without or a combination of the two. 

There are many methods that may be used to carry out an evaluation, Tfie Program Review Process 
has been presented as a straightforward and adaptable method which can be modified to meet a 
diversity of needs, ITie process presented is not meant to be a research method tested for reliability 
and validity. Instead it is designed to be a practical and useable tool to aid individuals and 
organizations in finding out more about their program. 

There are certainly limitatioriS to evaluation. One reality in tlie worid of evaluation is that of politics 
and hidden agendas. For this reason understanding and making explicit the real purpose of an 
evaluation is paramount to its effectiveness. This is not always possible and in such cases evaluation 
may be no more than a show piece that serves the interests of one particular stakeholder. Other 
evaluation limitations include lack of money^ lack of time, poor timing, poor collaboration with 
stakeholders during the evaluation desi/jn, terms of reference that are too broad and ineffective 
evaluators. However, the Program Review Process is designed to help you construct and conduct 
an evaluation that fits your needs and resources. When used carefully it will greatly facilitate a well 
designed and conducted review. Good luck in your efforts. 



AEE 20th International Conference - Proceedings Manual 



page 258 



References 

Braverman,M. (1990). Tliree approaches to evaluation: A ro[)es course illustration. Journal of 
Experiential Education. 13, (1), 23 - 30. 

Beiinet, D. B. (1989). Four steps to evaluating environmental education learning experiences. 
Journal of Environmental Education. 20, (2), 14 - 21. 

Flor, R. (1991). An introduction to research and evaluation in practice. Jounutl of Experiential 
Education. 14, (1), 36 - 39. 

Hamilton, S. F. (1980). Evaluating your own program. Educational Leadership. (4), 14 - 19. 

Harper, S. (1988). A model for program evaluation. Education Canada. 28, (3), 18 - 23. 

Kolb, D. G. (1991). Meaningful methods: Evaluation without the crunch. Journal of Experiential 
Education. 14, (1), 40 - 44. 

Popham, W. J. (1988). Educational Evaluation. Prentice Hall, Englewood Cliffs, NJ. 

Sergiovanni, T. J. (1987). The Principalship: A reflective practice perspective, Allyn and Bacon Inc. 
Toronto. 



AEE 20th Inltmationat Conference - Proceedings Manual page 259 




'^2 



